Bounded Loss Classification∗

Consider the problem of classification. Modern-day solutions have looked toward problem formulations where the search space is convex. Such formulations guarantee that a minimization of the objective function is found. But, in order to achieve that guarantee, such formulations treat outliers somewhat overzealously. Many classification objectives can be viewed as minimizing a loss funciton. For example, the Support Vector Machine minimizes the hinge loss, ∑ i lh(w T xi + b), where lh(z) = (1 − z)+. Logistic regression (LR) minimizes a similar loss, the logistic, ∑ i ll(w T xi + b), where ll(z) = − log 1 1+e−x . Many would not call these two losses “similar,” but they share important properties. One is that they are convex, e.g. lh(αz1 + (1− α)z2) ≥ αlh(z1) + (1− α)lh(z2). This is the reason that the objective has a unique minimum. Along with this property comes a less desirable property, that the loss for a single example is unbounded. In other words, an outlier can have an unbounded effect on the decision boundary. In practice, regularization is used to temper this effect, but it can produce negative effects. In summary, state-of-the-art classifiers utilize a convex loss function to achieve an objective with a unique minimum, but as a result, outliers can significantly effect the decision boundary. Motivation for current, state-of-the-art techniques was a long history of classification algorithms that used non-convex objective functions. For classification, one wishes to learn a decision boundary that will minimize the zero-one loss (lz(x) = θ(−wT x + b), θ is the heavaside function) on unseen examples drawn from the same distribution as the training examples. Many objectives minimized this or something very similar. Of course, such an optimization is riddled with local minima. Various techniques were developed to work around this problem, but none was as effective as the convex objectives that have been recently brought forward. We believe there is still hope in a more traditional objective, one that minimizes the zero-one loss. There are issues to be addressed—since the zero-one loss is not convex, how can we find a good solution? But, we feel that there are ways to sufficiently address this issue. Additionally, with the ∗Joint work with Tommi Jaakkola, [email protected]